Theoretical and Experimental Evaluation of Subspace Information Criterion

Recently, a new model selection criterion called the subspace information criterion (SIC) was proposed. SIC works well with small samples since it gives an unbiased estimate of the generalization error with finite samples. In this paper, we theoretically and experimentally evaluate the effectiveness of SIC in comparison with existing model selection techniques including the traditional leave-one-out cross-validation (CV), Mallows’s CP , Akaike’s information criterion (AIC), Sugiura’s corrected AIC (cAIC), Schwarz’s Bayesian information criterion (BIC), Rissanen’s minimum description length criterion (MDL), and Vapnik’s measure (VM). Theoretical evaluation includes the comparison of the generalization measure, approximation method, and restriction on model candidates and learning methods. Experimentally, the performance of SIC in various situations is investigated. The simulations show that SIC outperforms existing techniques especially when the number of training examples is small and the noise variance is large.